Github News

• 

  Behind the scenes of GitHub Token Scanning
  • sorangutan  

  1
  1
  Posts
  38
  Views

  

  

  Several years ago we started scanning all pushes to public repositories for GitHub OAuth tokens and personal access tokens. Now we’re extending this capability to include tokens from cloud service providers and additional credentials, such as unencrypted SSH private keys associated with a user’s GitHub account.

  We live in amazing times for software development. Capabilities that were once only available to large technology companies are now accessible to the smallest of startups. Developers can leverage cloud services to quickly perform continuous integration testing, deploy their code to fully scalable infrastructure, accept credit card payments from customers, and nearly anything else you can imagine.

  Composing cloud services like this is the norm going forward, but it comes with inherent security complexities. Each cloud service a developer typically uses requires one or more credentials, often in the form of API tokens. In the wrong hands, they can be used to access sensitive customer data—or vast computing resources for mining cryptocurrency, presenting significant risks to both users and cloud service providers.

  diff --git a/config.yml b/config.yml
  index 1110370..da7b5de 100644
  --- a/config.yml
  +++ b/config.yml
  @@ -1,2 +1,2 @@
   production:
  -  github_token: 4e862a8ec12ef0ad7c1337e8b16d98f4d764b8f6
  +  github_token: <%= ENV["GITHUB_TOKEN"] %>
  

  ^{Fig 1: Your master branch might look innocent, but exposed credentials in your project’s history could prove costly.}

  Token Scanning 1.0

  GitHub, and our users, aren’t immune to this problem. That is why we started scanning for GitHub OAuth tokens in public repositories several years ago. But, our users are not just GitHub customers. They are also customers of cloud infrastructure providers, cloud payment processing providers, and other cloud service providers that have become commonplace in modern development.

  Our existing solution focused on GitHub OAuth tokens exclusively and was never designed for extensibility. The existing code leveraged hand-tuned assembly that was extremely fast at finding 40-hex character strings (the format of GitHub OAuth tokens). This bit of code was patched into Git and run inline whenever code was pushed to GitHub. It was an amazing piece of work, but could not support multiple credential formats. Our vision was to support all of the popular cloud service providers.

  Token Scanning 2.0

  So began our journey into the next generation of GitHub Token Scanning. The obvious path to a more extensible scanner is some form of regular expression support. However, scanning for credentials using typical regular expression libraries doesn’t scale from a performance perspective, as they optimize for a slightly different problem than the one we have.

  The vast majority of regular expression libraries are designed to return the first match in a set of patterns. Given we have at least one pattern for each cloud service provider, we require all matches are returned and not just the first. The only way to ensure this with traditional libraries is to scan a given input once for each pattern. However, this increases the scan time dramatically for large repositories or large sets of patterns. Fortunately, scanning Git data for credentials is just a specific case of a general problem. For example, high-performance application-level firewalls similarly need to scan high-volume network traffic for sets of patterns to identify known viruses or malware. If you squint, scanning high-volume Git push data for credentials is a very similar problem.

  Our research eventually lead us to a GitHub repository hosting the amazing Hyperscan library by Intel. This library is incredibly performant and provides exactly what we need. We will explore the technical details in more depth in a follow-up engineering post. But, in short, Hyperscan let us replace all of the assembly code patches to Git with a new standalone scanner, written in Go, that has scaled nicely.

  In parallel with working on the implementation, we reached out to several cloud service providers we thought would be interested in testing out Token Scanning in a private beta. They were all enthusiastic to participate, as many of them had contacted us in the past looking for a solution to this widespread problem.

  Since April, we’ve worked with cloud service providers in private beta to scan all changes to public repositories and public Gists for credentials (GitHub doesn’t scan private code). Each candidate credential is sent to the provider, including some basic metadata such as the repository name and the commit that introduced the credential. The provider can then validate the credential and decide if the credential should be revoked depending on the associated risks to the user or provider. Either way, the provider typically contacts the owner of the credential, letting them know what occurred and what action was taken.

  Where we go from here

  We have received amazing feedback from both providers and users during the private beta. Cloud service providers have told us that GitHub Token Scanning has been tremendously effective in helping them identify credentials before malicious users. And, while GitHub users were not aware that Token Scanning was in beta, we did take notice of a tweet from a GitHub user that included a number of enthusiastic exclamation marks and 🔥 emojis. This user was extremely grateful for having received a notification from a participating cloud service provider less than a minute after they had accidentally pushed a highly sensitive credential to a public repository.

  During the beta we have scanned millions of public repository changes and identified millions of candidate credentials. As announced yesterday at GitHub Universe, Token Scanning is now in public beta, and supports an increasing number of cloud providers. We’re excited by the impact of Token Scanning today and have lots of ideas about how to make it even more powerful in the future. Dealing with credentials is an unavoidable part of modern development. With GitHub by your side, we hope to minimize the security impact of such accidents.

  
  
  https://blog.github.com/2018-10-17-behind-the-scenes-of-github-token-scanning/
• 

  Applying machine intelligence to GitHub security alerts
  • sorangutan  

  1
  1
  Posts
  56
  Views

  

  Last year, we released security alerts that track security vulnerabilities in Ruby and JavaScript packages. Since then, we’ve identified more than four million of these vulnerabilities and added support for Python. In our launch post, we mentioned that all vulnerabilities with CVE IDs are included in security alerts, but sometimes there are vulnerabilities that are not disclosed in the National Vulnerability Database. Fortunately, our collection of security alerts can be supplemented with vulnerabilities detected from activity within our developer community.

  Leveraging the community

  There are many places a project can publicize security fixes within a new version: the CVE feed, various mailing lists, and open source groups, or even within its release notes or changelog. Regardless of how projects share this information, some developers within the GitHub community will see the advisory and immediately bump their required versions of the dependency to a known safe version. If detected, we can use the information in these commits to generate security alerts for vulnerabilities which may not have been published in the CVE feed.

  On an average day, the dependency graph can track around 10,000 commits to dependency files for any of our supported languages. We can’t manually process this many commits. Instead, we depend on machine intelligence to sift through them and extract those that might be related to a security release.

  For this purpose, we created a machine learning model that scans text associated with public commits (the commit message and linked issues or pull requests) to filter out those related to possible security upgrades. With this smaller batch of commits, the model uses the diff to understand how required version ranges have changed. Then it aggregates across a specific timeframe to get a holistic view of all dependencies that a security release might affect. Finally, the model outputs a list of packages and version ranges it thinks require an alert and currently aren’t covered by any known CVE in our system.

  Always quality focused

  No machine learning model is perfect. While machine intelligence can sift through thousands of commits in an instant, this anomaly-detection algorithm will still generate false positives for packages where no security patch was released. Security alert quality is a focus for us, so we review all model output before the community receives an alert.

  Learn more

  Interested in learning more? Join us at GitHub Universe next week to explore the connections that push technology forward and keep projects secure through talks, trainings, and workshops. Tune in to the blog October 16-17 for more updates and announcements.

  The post Applying machine intelligence to GitHub security alerts appeared first on The GitHub Blog.

  
  
  https://github.blog/2018-10-09-applying-machine-intelligence-to-security-alerts/
• 

  Applying machine intelligence to GitHub security alerts
  • sorangutan  

  1
  1
  Posts
  90
  Views

  

  

  Last year, we released security alerts that track security vulnerabilities in Ruby and JavaScript packages. Since then, we’ve identified more than four million of these vulnerabilities and added support for Python. In our launch post, we mentioned that all vulnerabilities with CVE IDs are included in security alerts, but sometimes there are vulnerabilities that are not disclosed in the National Vulnerability Database. Fortunately, our collection of security alerts can be supplemented with vulnerabilities detected from activity within our developer community.

  Leveraging the community

  There are many places a project can publicize security fixes within a new version: the CVE feed, various mailing lists and open source groups, or even within its release notes or changelog. Regardless of how projects share this information, some developers within the GitHub community will see the advisory and immediately bump their required versions of the dependency to a known safe version. If detected, we can use the information in these commits to generate security alerts for vulnerabilities which may not have been published in the CVE feed.

  On an average day, the dependency graph can track around 10,000 commits to dependency files for any of our supported languages. We can’t manually process this many commits. Instead, we depend on machine intelligence to sift through them and extract those that might be related to a security release.

  For this purpose, we created a machine learning model that scans text associated with public commits (the commit message and linked issues or pull requests) to filter out those related to possible security upgrades. With this smaller batch of commits, the model uses the diff to understand how required version ranges have changed. Then it aggregates across a specific timeframe to get a holistic view of all dependencies that a security release might affect. Finally, the model outputs a list of packages and version ranges it thinks require an alert and currently aren’t covered by any known CVE in our system.

  Always quality focused

  No machine learning model is perfect. While machine intelligence can sift through thousands of commits in an instant, this anomaly-detection algorithm will still generate false positives for packages where no security patch was released. Security alert quality is a focus for us, so we review all model output before the community receives an alert.

  Learn more

  Interested in learning more? Join us at GitHub Universe next week to explore the connections that push technology forward and keep projects secure through talks, trainings, and workshops. Tune in to the blog October 16-17 for more updates and announcements.

  
  
  https://blog.github.com/2018-10-09-applying-machine-intelligence-to-security-alerts/
• 

  Applying machine intelligence to GitHub security alerts
  • sorangutan  

  1
  1
  Posts
  36
  Views

  

  

  Last year, we released security alerts that track security vulnerabilities in Ruby and JavaScript packages. Since then, we’ve identified more than four million of these vulnerabilities and added support for Python. In our launch post, we mentioned that all vulnerabilities with CVE IDs are included in security alerts, but sometimes there are vulnerabilities that are not disclosed in the National Vulnerability Database. Fortunately, our collection of security alerts can be supplemented with vulnerabilities detected from activity within our developer community.

  Leveraging the community

  There are many places a project can publicize security fixes within a new version: the CVE feed, various mailing lists and open source groups, or even within its release notes or changelog. Regardless of how projects share this information, some developers within the GitHub community will see the advisory and immediately bump their required versions of the dependency to a known safe version. If detected, we can use the information in these commits to generate security alerts for vulnerabilities which may not have been published in the CVE feed.

  On an average day, the dependency graph can track around 10,000 commits to dependency files for any of our supported languages. We can’t manually process this many commits. Instead, we depend on machine intelligence to sift through them and extract those that might be related to a security release.

  For this purpose, we created a machine learning model that scans text associated with public commits (the commit message and linked issues or pull requests) to filter out those related to possible security upgrades. With this smaller batch of commits, the model uses the diff to understand how required version ranges have changed. Then it aggregates across a specific timeframe to get a holistic view of all dependencies that a security release might affect. Finally, the model outputs a list of packages and version ranges it thinks require an alert and currently aren’t covered by any known CVE in our system.

  Always quality focused

  No machine learning model is perfect. While machine intelligence can sift through thousands of commits in an instant, this anomaly-detection algorithm will still generate false positives for packages where no security patch was released. Security alert quality is a focus for us, so we review all model output before the community receives an alert.

  Learn more

  Interested in learning more? Join us at GitHub Universe next week to explore the connections that push technology forward and keep projects secure through talks, trainings, and workshops. Tune in to the blog October 16-17 for more updates and announcements.

  
  
  https://blog.github.com/2018-10-09-applying-machine-intelligence-to-security-alerts/
• 

  Git Submodule Vulnerability Announced
  • sorangutan  

  1
  1
  Posts
  68
  Views

  

  The Git project has disclosed CVE-2018-17456, a vulnerability in Git that can cause arbitrary code to be executed when a user clones a malicious repository. Git v2.19.1 has been released with a fix, along with backports in v2.14.5, v2.15.3, v2.16.5, v2.17.2, and v2.18.1. We encourage all users to update their clients to protect themselves.

  Until you’ve updated, you can protect yourself by avoiding submodules from untrusted repositories. This includes commands such as git clone --recurse-submodules and git submodule update.

  Affected products

  GitHub Desktop

  GitHub Desktop versions 1.4.1 and older included an embedded version of Git that was affected by this vulnerability. We encourage all GitHub Desktop users to update to the newest version (1.4.2 and 1.4.3-beta0) available today in the Desktop app.

  Atom

  Atom included the same embedded Git and was also affected. Releases 1.31.2 and 1.32.0-beta3 include the patch.

  Ensure you’re on the latest Atom release by completing any of the following:

  • Windows: From the toolbar, click Help -> Check for Updates
  • MacOS: From the menu bar, click Atom -> Check for Update
  • Linux: Update manually by downloading the latest release from atom.io

  Git on the command line and other clients

  In order to be protected from the vulnerability, you must update your command-line version of Git, and any other application that may include an embedded version of Git, as they are independent of each other.

  Additional notes

  Neither GitHub.com nor GitHub Enterprise are directly affected by the vulnerability. However, as with previously discovered vulnerabilities, GitHub.com will detect malicious repositories, and will reject pushes or API requests attempting to create them. Versions of GitHub Enterprise with this detection will ship on October 9.

  Details of the vulnerability

  This vulnerability is very similar to CVE-2017-1000117, as both are option-injection attacks related to submodules. In the earlier attack, a malicious repository would ship a .gitmodules file pointing one of its submodules to a remote repository with an SSH host starting with a dash (-). The ssh program—spawned by Git—would then interpret that as an option. This attack works in a similar way, except that the option-injection is against the child git clone itself.

  The problem was reported on September 23 by @joernchen, both to Git’s private security list, as well as to GitHub’s Bug Bounty program. Developers at GitHub worked with the Git community to develop a fix.

  The basic fix was clear from the report. However, due to to the similarity to CVE-2017-1000117, we also audited all of the .gitmodules values and implemented stricter checks as appropriate. These checks should prevent a similar vulnerability in another code path. We also implemented detection of potentially malicious submodules as part of Git’s object quality checks (which was made much easier by the infrastructure added during the last submodule-related vulnerability).

  The coordinated disclosure date of October 5 was selected by Git developers to allow packagers to prepare for the release. This also provided hosting sites (with custom implementations) ample time to detect and block the attack before it became public. Members of the Git community checked the JGit and libgit2 implementations. Those are not affected by the vulnerability because they clone submodules via function calls rather than separate commands.

  We were also able to use the time to scan all repositories on GitHub for evidence of the attack being used in the wild. We’re happy to report that no instances were found (and now, with our detection, none can be added).

  Please update your copy of Git soon, and happy cloning!

  
  
  https://blog.github.com/2018-10-05-git-submodule-vulnerability/
• 

  Git Submodule Vulnerability Announced
  • sorangutan  

  1
  1
  Posts
  31
  Views

  

  The Git project has disclosed CVE-2018-17456, a vulnerability in Git that can cause arbitrary code to be executed when a user clones a malicious repository. Git v2.19.1 has been released with a fix, along with backports in v2.14.5, v2.15.3, v2.16.5, v2.17.2, and v2.18.1. We encourage all users to update their clients to protect themselves.

  Until you’ve updated, you can protect yourself by avoiding submodules from untrusted repositories. This includes commands such as git clone --recurse-submodules and git submodule update.

  Affected products

  GitHub Desktop

  GitHub Desktop versions 1.4.1 and older included an embedded version of Git that was affected by this vulnerability. We encourage all GitHub Desktop users to update to the newest version (1.4.2 and 1.4.3-beta0) available today in the Desktop app.

  Atom

  Atom included the same embedded Git and was also affected. Releases 1.31.2 and 1.32.0-beta3 include the patch.

  Ensure you’re on the latest Atom release by completing any of the following:

  • Windows: From the toolbar, click Help -> Check for Updates
  • MacOS: From the menu bar, click Atom -> Check for Update
  • Linux: Update manually by downloading the latest release from atom.io

  Git on the command line and other clients

  In order to be protected from the vulnerability, you must update your command-line version of Git, and any other application that may include an embedded version of Git, as they are independent of each other.

  Additional notes

  Neither GitHub.com nor GitHub Enterprise are directly affected by the vulnerability. However, as with previously discovered vulnerabilities, GitHub.com will detect malicious repositories, and will reject pushes or API requests attempting to create them. Versions of GitHub Enterprise with this detection will ship on October 9.

  Details of the vulnerability

  This vulnerability is very similar to CVE-2017-1000117, as both are option-injection attacks related to submodules. In the earlier attack, a malicious repository would ship a .gitmodules file pointing one of its submodules to a remote repository with an SSH host starting with a dash (-). The ssh program—spawned by Git—would then interpret that as an option. This attack works in a similar way, except that the option-injection is against the child git clone itself.

  The problem was reported on September 23 by @joernchen, both to Git’s private security list, as well as to GitHub’s Bug Bounty program. Developers at GitHub worked with the Git community to develop a fix.

  The basic fix was clear from the report. However, due to to the similarity to CVE-2017-1000117, we also audited all of the .gitmodules values and implemented stricter checks as appropriate. These checks should prevent a similar vulnerability in another code path. We also implemented detection of potentially malicious submodules as part of Git’s object quality checks (which was made much easier by the infrastructure added during the last submodule-related vulnerability).

  The coordinated disclosure date of October 5 was selected by Git developers to allow packagers to prepare for the release. This also provided hosting sites (with custom implementations) ample time to detect and block the attack before it became public. Members of the Git community checked the JGit and libgit2 implementations. Those are not affected by the vulnerability because they clone submodules via function calls rather than separate commands.

  We were also able to use the time to scan all repositories on GitHub for evidence of the attack being used in the wild. We’re happy to report that no instances were found (and now, with our detection, none can be added).

  Please update your copy of Git soon, and happy cloning!

  
  
  https://blog.github.com/2018-10-05-git-submodule-vulnerability/
• 

  Upgrading GitHub from Rails 3.2 to 5.2
  • sorangutan  

  1
  1
  Posts
  32
  Views

  

  On August 15th GitHub celebrated a major milestone: our main application is now running on the latest version of Rails: 5.2.1!

  In total the project took a year and a half to upgrade from Rails 3.2 to Rails 5.2. Along the way we took time to clean up technical debt and improve the overall codebase while doing the upgrade. Below we’ll talk about how we upgraded Rails, lessons we learned and whether we’d do it again.

  How did we do it?

  Upgrading Rails on an application as large and as trafficked as GitHub is no small task. It takes careful planning, good organization, and patience. The upgrade started out as kind of a hobby; engineers would work on it when they had free time. There was no dedicated team. As we made progress and gained traction it became not only something we hoped we could do, but a priority.

  Since GitHub is so important to our community, we can’t stop feature development or bug fixes in order to upgrade Rails.

  Instead of using a long-running branch to upgrade Rails we added the ability to dual boot the application in multiple versions of Rails. We created two Gemfile.lock’s: one for the current version Gemfile.lock and one for the future version Gemfile_next.lock. The dual booting allows us to regularly deploy changes for the next version to GitHub without requiring long running branches or altering how production works. We do this by conditionally loading the code.

  if GitHub.rails_3_2?
    ## 3.2 code (i.e. production a year and a half ago)
  elsif GitHub.rails_4_2?
    # 4.2 code
  else
    # all 5.0+ future code, ensuring we never accidentally
    # fall back into an old version going forward
  end

  Each time we got a minor version of Rails green we’d make the CI job required for all pushes to the GitHub application and start work on the next version. While we worked on Rails 4.1 a CI job would run on every push for 3.2 and 4.0. When 4.1 was green we’d swap out 4.1 for 4.0 and get to work on 4.2. This allowed us to prevent regressions once a version of Rails was green, and time for engineers to get used to writing code that worked in multiple versions of Rails.

  The two versions that we deployed were 4.2 and 5.2. We deployed 4.2 because it was a huge milestone and was the first version of Rails that hadn’t been EOL’d yet (as an aside: we’d been backporting security fixes to 3.2 but not to 4.0+ so we couldn’t deploy 4.0 or 4.1. Rest assured your security is our top priority).

  To roll out the Rails upgrade we created a careful and iterative process. We’d first deploy to our testing environment and requested volunteers from each team to click test their area of the codebase to find any regressions the test suite missed.

  After fixing those regressions, we deployed in off-hours to a percentage of production servers. During each deploy we’d collect data about exceptions and performance of the site. With that information we’d fix bugs that came up and repeat those steps until the error rate was low enough be considered equal to the previous version. We merged the upgrade once we could deploy to full production for 30 minutes at peak traffic with no visible impact.

  This process allowed us to deploy 4.2 and 5.2 with minimal customer impact and no down time.

  Lessons Learned

  The Rails upgrade took a year and a half. This was for a few reasons. First, Rails upgrades weren’t always smooth and some versions had major breaking changes. Rails improved the upgrade process for the 5 series so this meant that while 3.2 to 4.2 took 1 year, 4.2 to 5.2 only took 5 months.

  Another reason is the GitHub codebase is 10 years old. Over the years technical debt builds and there’s bound to be gremlins lurking in the codebase. If you’re on an old version of Rails, your engineers will need to add more monkey patches or implement features that exist upstream.

  Lastly, when we started it wasn’t clear what resources were needed to support the upgrade and since most of us had never done a Rails upgrade before, we were learning as we went. The project originally began with 1 full-time engineer and a small army of volunteers. We grew that team to 4 full-time engineers plus the volunteers. Each version bump meant we learned more and the next version went even faster.

  Through this work we learned some important lessons that we hope make your next upgrade easier:

  • Upgrade early and upgrade often. The closer you are to a new version of Rails, the easier upgrades will be. This encourages your team to fix bugs in Rails instead of monkey-patching the application or reinventing features that exist upstream.
  • Keep upgrade infrastructure in place. There will always be a new version to upgrade to, so once you’re on a modern version of Rails add a build to run against the master branch. This will catch bugs in Rails and your application early, make upgrades easier, and increase your upstream contributions.
  • Upstream your tooling instead of rolling your own. The more you push upstream to gems or Rails, the less logic you need in your application. Save your application code for what truly makes your company special (i.e. Pull Requests), instead of tools to make your application run smoothly (i.e. concurrent testing libraries)
  • Avoid using private API’s in your frameworks. Rails has a lot of code that’s not private but isn’t documented on purpose. That code is subject to change without notice, so writing code that relies on private code can easily break in an upgrade.
  • Address technical debt often. It’s easy to think “this is working, why mess with it”, but if no one knows how that code works, it can quickly become a bottleneck for upgrades. Try to prevent coupling your application logic too closely to your framework. Ensure that the line where your application ends and your framework begins is clear. You can do this by addressing technical debt before it becomes difficult to remove.
  • Do incremental upgrades. Each minor version of Rails provides the deprecation warnings for the next version. By upgrading from 3.2 to 4.0, 4.0 to 4.1, etc we were able to identify problems in the next version early and define clear milestones.
  • Keep up the momentum. Rails upgrades can seem daunting. Create ways in which your team can have quick wins to keep momentum going. Share the responsibility across teams so that everyone is familiar with the new version of the framework and prevent burnout. Once you’re on the newest version add a build to your app that periodically runs your suite against edge Rails so you can catch bugs in your code or your framework early.
  • Expect things to break. Upgrades are hard and in an application as large as GitHub things are bound to break. While we didn’t take the site down during the upgrade we had issues with CI, local development, slow queries, and other problems that didn’t show up in our CI builds or click testing.

  Was it worth it?

  Absolutely.

  Upgrading Rails has allowed us to address technical debt in our application. Our test suite is now closer to vanilla Rails, we were able to remove StateMachine in favor of Active Record enums, and start replacing our job runner with Active Job. And that’s just the beginning.

  Rails upgrades are a lot of hard work and can be time-consuming, but they also open up a ton possibilities. We can push more of our tooling upstream to Rails, address areas of technical debt, and be one of the largest codebases running on the most recent version of Rails. Not only does this benefit us at GitHub, it benefits our customers and the open source community.

  The post Upgrading GitHub from Rails 3.2 to 5.2 appeared first on The GitHub Blog.

  
  
  https://github.blog/2018-09-28-upgrading-github-from-rails-3-2-to-5-2/
• 

  Introducing Experiments, an ongoing research effort from GitHub
  • sorangutan  

  1
  1
  Posts
  58
  Views

  

  Software developers are most productive when software development is inclusive and accessible. At GitHub, we conduct research in machine learning, design, and infrastructure to make sure everyone can do their best work with the next generation of developer tools and workflows.

  This research can take considerable time to reach you, our end users, if it reaches you at all. We rigorously evaluate products for stability, performance, and security. And many experiments don’t meet our success criteria for product release, even when they present a path forward for future innovation.

  Introducing Experiments

  Although we can’t share everything we do, we’ve launched a collection of demonstrations highlighting our most exciting research projects—and the ideas behind them—with Experiments. We hope these will not only give you insight into our research but inspire you to think audaciously about the future of software development.

  See our first experiment

  For our first demo, we’ve chosen Semantic Code Search. We’ve used machine learning to build semantic representations of code that allow you to use natural language to search for code by intent, rather than just keyword matching. See our blog post for additional detail on how this works.

  

  We’re just getting started, so stay tuned for more examples. If this research excite you as much as they excite us, why not join our team?

  The post Introducing Experiments, an ongoing research effort from GitHub appeared first on The GitHub Blog.

  
  
  https://github.blog/2018-09-18-introducing-experiments-an-ongoing-research-effort-from-github/
• 

  Towards Natural Language Semantic Code Search
  • sorangutan  

  1
  1
  Posts
  54
  Views

  

  

  This blog post complements a live demonstration on our recently announced site: experiments.github.com

  Motivation

  Searching code on GitHub is currently limited to keyword search. This assumes either the user knows the syntax, or can anticipate what keywords might be in comments surrounding the code they are looking for. Our machine learning scientists have been researching ways to enable the semantic search of code.

  To fully grasp the concept of semantic search, consider the below search query, “ping REST api and return results”:

  

  Note that the demonstrated semantic search returns reasonable results even though there are no keywords in common between the search query and the text (the code & comments found do not contain the words “Ping”, “REST” or “api”)! The implications of augmenting keyword search with semantic search are profound. For example, such a capability would expedite the process of on-boarding new software engineers onto projects and bolster the discoverability of code in general.

  In this post, we want to share how we are leveraging deep learning to make progress towards this goal. We also share an open source example with code and data that you can use to reproduce these results!

  Introduction

  One of the key areas of machine learning research underway at GitHub is representation learning of entities, such as repos, code, issues, profiles and users. We have made significant progress towards enabling semantic search by learning representations of code that share a common vector space as text. For example, consider the below diagram:

  

  In the above example, Text 2 (blue) is a reasonable description of the code, whereas Text 1 (red) is not related to the code at all. Our goal is to learn representations where (text, code) pairs that describe the same concept are close neighbors, whereas unrelated (text, code) pairs are further apart. By representing text and code in the same vector space, we can vectorize a user’s search query and lookup the nearest neighbor that represents code. Below is a four-part description of the approach we are currently using to accomplish this task:

  1. Learning Representations of Code

  In order to learn a representation of code, we train a sequence-to-sequence model that learns to summarize code. A way to accomplish this for Python is to supply (code, docstring) pairs where the docstring is the target variable the model is trying to predict. One active area of research for us is incorporating domain specific optimizations like tree-based LSTMs, gated-graph networks and syntax-aware tokenization. Below is a screenshot that showcases the code summarizer model at work. In this example, there are two python functions supplied as input, and in both cases the model produces a reasonable summary of the code as output:

  

  It should be noted that in the above examples, the model produces the summary by using the entire code blob, not merely the function name.

  Building a code summarizer is a very exciting project on its own, however, we can utilize the encoder from this model as a general purpose feature extractor for code. After extracting the encoder from this model, we can fine-tune it for the task of mapping code to the vector space of natural language.

  We can evaluate this model objectively using the BLEU score. Currently we have been able to achieve a BLEU score of 13.5 on a holdout set of python code, using the fairseq-py library for sequence to sequence models.

  2. Learning Representations of Text Phrases

  In addition to learning a representation for code, we needed to find a suitable representation for short phrases (like sentences found in Python docstrings). Initially, we experimented with the Universal Sentence Encoder, a pre-trained encoder for text that is available on TensorFlow Hub. While the embeddings from worked reasonably well, we found that it was advantageous to learn embeddings that were specific to the vocabulary and semantics of software development. One area of ongoing research involves evaluating different domain-specific corpuses for training our own model, ranging from GitHub issues to third party datasets.

  To learn this representation of phrases, we trained a neural language model by leveraging the fast.ai library. This library gave us easy access to state of the art architectures such as AWD LSTMs, and to techniques such as cyclical learning rates with random restarts. We extracted representations of phrases from this model by summarizing the hidden states using the concat pooling approach found in this paper.

  One of the most challenging aspects of this exercise was to evaluate the quality of these embeddings. We are currently building a variety of downstream supervised tasks similar to those outlined here that will aid us in evaluating the quality of these embeddings objectively. In the meantime, we sanity check our embeddings by manually examining the similarity between similar phrases. The below screenshot illustrates examples where we search the vectorized docstrings for similarity against user-supplied phrases:

  

  3. Mapping Code Representations To The Same Vector-Space as Text

  Next, we map the code representations we learned from the code summarization model (part 1) to the vector space of text. We accomplish this by fine-tuning the encoder of this model. The inputs to this model are still code blobs, however the target variable the model is now the vectorized version of docstrings. These docstrings are vectorized using the approach discussed in the previous section.

  Concretely, we perform multi-dimensional regression with cosine proximity loss to bring the hidden state of the encoder into the same vector-space as text.

  We are actively researching alternate approaches that directly learn a joint vector space of code and natural language, borrowing from some ideas outlined here.

  4. Creating a Semantic Search System

  Finally, after successfully creating a model that can vectorize code into the same vector-space as text, we can create a semantic search mechanism. In its most simple form, we can store the vectorized version of all code in a database, and perform nearest neighbor lookups to a vectorized search query.

  Another active area of our research is determining the best way to augment existing keyword search with semantic results and how to incorporate additional information such as context and relevance.
  Furthermore, we are actively exploring ways to evaluate the quality of search results that will allow us to iterate quickly on this problem. We leave these topics for discussion in a future blog post.

  Summary

  The below diagram summarizes all the steps in our current semantic-search workflow:

  

  We are exploring ways to improve almost every component of this approach, including data preparation, model architecture, evaluation procedures, and overall system design. What is described in this blog post is only a minimal example that scratches the surface.

  Open Source Examples

  Our open-source end-to-end tutorial contains a detailed walkthrough of the approach outlined in this blog, along with code and data you can use to reproduce the results.

  This open source example (with some modifications) is also used as a tutorial for the kubeflow project, which is implemented here.

  Limitations and Intended Use Case(s)

  We believe that semantic code search will be most useful for targeted searches of code within specific entities such as repos, organizations or users as opposed to general purpose “how to” queries. The live demonstration of semantic code search hosted on our recently announced Experiments site does not allow users to perform targeted searches of repos. Instead, this demonstration is designed to share a taste of what might be possible and searches only a limited, static set of python code.

  Furthermore, like all machine learning techniques, the efficacy of this approach is limited by the training data used. For example, the data used to train these models are (code, docstring) pairs. Therefore, search queries that closely resemble a docstring have the greatest chance of success. On the other hand, queries that do not resemble a docstring or contain concepts for which there is little data may not yield sensible results. Therefore, it is not difficult to challenge our live demonstration and discover the limitations of this approach. Nevertheless, our initial results indicate that this is an extremely fruitful area of research that we are excited to share with you.

  There are many more use cases for semantic code search. For example, we could extend the ideas presented here to allow users to search for code using the language of their choice (French, Mandarin, Arabic, etc.) across many different programming languages simultaneously.

  Get In Touch

  This is an exciting time for the machine learning research team at GitHub and we are looking to expand. If our work interests you, please get in touch!

  Contributors

  

  hamelsmu

  ---

  

  hohsiangwu

  The post Towards Natural Language Semantic Code Search appeared first on The GitHub Blog.

  
  
  https://github.blog/2018-09-18-towards-natural-language-semantic-code-search/
• 

  Introducing Experiments, an ongoing research effort from GitHub
  • sorangutan  

  1
  1
  Posts
  70
  Views

  

  Software developers are most productive when software development is inclusive and accessible. At GitHub, we conduct research in machine learning, design, and infrastructure to make sure everyone can do their best work with the next generation of developer tools and workflows.

  This research can take considerable time to reach you, our end users, if it reaches you at all. We rigorously evaluate products for stability, performance, and security. And many experiments don’t meet our success criteria for product release, even when they present a path forward for future innovation.

  Introducing Experiments

  Although we can’t share everything we do, we’ve launched a collection of demonstrations highlighting our most exciting research projects—and the ideas behind them—with Experiments. We hope these will not only give you insight into our research but inspire you to think audaciously about the future of software development.

  See our first experiment

  For our first demo, we’ve chosen Semantic Code Search. We’ve used machine learning to build semantic representations of code that allow you to use natural language to search for code by intent, rather than just keyword matching. See our blog post for additional detail on how this works.

  

  We’re just getting started, so stay tuned for more examples. If this research excite you as much as they excite us, why not join our team?

  
  
  https://blog.github.com/2018-09-18-introducing-experiments-an-ongoing-research-effort-from-github/
• 

  Introducing Experiments, an ongoing research effort from GitHub
  • sorangutan  

  1
  1
  Posts
  26
  Views

  

  Software developers are most productive when software development is inclusive and accessible. At GitHub, we conduct research in machine learning, design, and infrastructure to make sure everyone can do their best work with the next generation of developer tools and workflows.

  This research can take considerable time to reach you, our end users, if it reaches you at all. We rigorously evaluate products for stability, performance, and security. And many experiments don’t meet our success criteria for product release, even when they present a path forward for future innovation.

  Introducing Experiments

  Although we can’t share everything we do, we’ve launched a collection of demonstrations highlighting our most exciting research projects—and the ideas behind them—with Experiments. We hope these will not only give you insight into our research but inspire you to think audaciously about the future of software development.

  See our first experiment

  For our first demo, we’ve chosen Semantic Code Search. We’ve used machine learning to build semantic representations of code that allow you to use natural language to search for code by intent, rather than just keyword matching. See our blog post for additional detail on how this works.

  

  We’re just getting started, so stay tuned for more examples. If this research excite you as much as they excite us, why not join our team?

  
  
  https://blog.github.com/2018-09-18-introducing-experiments-an-ongoing-research-effort-from-github/
• 

  Life as a GSoC student: What I learned about open source development through Probot
  • sorangutan  

  1
  1
  Posts
  90
  Views

  

  Abhijeet Pratap Singh is a student at the Indian Institute of Information Technology in Tiruchirappalli, India. He was selected to be one of the Google Summer of Code (GSoC) students for the Probot project. In this post, Abhijeet recounts his experience working with the Probot Team at GitHub and what he learned about working with other developers on an open source project.

  This spring, I was selected to participate in Google Summer of Code (GSoC), a program that pairs student developers with open source projects. I was placed with the Probot Team at GitHub. Probot is an open source framework used to build applications for GitHub based on NodeJS. These applications improve and automate your workflow while working on GitHub.

  Getting started with Probot

  I’m a Computer Science and Engineering undergraduate at the Indian Institute of Information Technology. When I entered the Institute, I was very interested in knowing more about software. I explored different opportunities and found out about GSoC through Quora earlier this year.

  I learned about Probot while researching the different projects partnering with GSoC. I had some previous experience working with bots and NodeJS and decided to try out applications built with Probot. I found the Probot community, a Slack community of developers who use Probot, which pointed me to the Probot Summer of Code and Probot’s GSoC project ideas. The project idea list helped me identify what project I should submit a proposal for.

  I initially focused on the Twitter integration proposed in Probot’s ideas repo. After a few days of tinkering, I successfully developed my first Twitter integration bot. I recorded a screencast to log my progress with the bot and shared it with Brandon Keepers (@bkeepers), Jason Etcovitch (@JasonEtco), and Gregor Martynus (@gr2m).

  I later submitted proposals for the Twitter integration and Weekly Digest projects when the student application period started. My proposal for the Weekly Digest project was accepted, and I became the Student Developer for Probot. The results were announced on April 23. I was so happy to see that my project was accepted and was really excited for this summer. Gregor Martynus (@gr2m) and Wilhelm Klopp (@wilhelmklopp) were assigned as my mentors.

  Working on the Weekly Digest project

  GSoC kicked off in late April, and I spent more time exploring the community and its best practices. I also met my mentors over video conference. We discussed a plan to move forward with and implement our projects. Then we created the Weekly Digest repository in order to track progress.

  The coding round began in mid May. I started by opening a few issues and pull requests, making some commits, and exploring GitHub’s REST API and GitHub’s REST API client for NodeJS. My mentors and I met weekly to go over the work I did and help out whenever I got stuck. It was a great way to discuss best practices, standards, and tools that developers use.

  After working on the project, I got very close to a pre-release version of the app and released an alpha version of the Weekly Digest.

  Here’s a GIF of the Weekly Digest app in action:

  

  How the Weekly Digest app works

  When you install the Weekly Digest in your (or your organization’s) repository, it curates the following data and publishes it as an issue:

  • Issues created in the last week
    • Opened issues
    • Closed issues
    • Noisy issues
    • Liked issues
  • Pull requests opened, updated, or merged in the last week
    • Opened pull requests
    • Updated pull requests
    • Merged pull requests
  • Commits made in the master branch in the last week
  • Contributors adding contributions in the last week
  • Stargazers, or the fans of your repositories, who really loved your repo
  • Releases of the project you’re working on

  The app, as the name suggests, generates these digests and publishes them on a weekly basis, typically on a Sunday. You can change the default configuration of the app by adding a .github/weekly-digest.yml file in your GitHub repository, which allows you to configure the publish date and the specific information included in the digest.

  Install the Weekly Digest app

  What I gained from this experience

  Becoming a software developer has been my dream ever since I was a kid. My mentors were happy to answer all of my questions and took time out of their busy schedules to meet with me. Through GSoC, I learned how a project is maintained and how developers work together to deliver solutions. And I got to develop a love for open source through my work with Probot!

  I was really lucky to be guided by my mentors, Gregor Martynus (@gr2m) and Wilhelm Klopp (@wilhelmklopp). Special thanks to them, Brandon Keepers (@bkeepers), Jason Etcovitch (@JasonEtco), and the awesome Probot community for accepting me as a Student Developer for this year’s GSoC!

  Learn more about Probot

  
  
  https://blog.github.com/2018-08-21-what-i-learned-about-open-source-development-from-probot/
• 

  Using Figma designs to build the Octicons icon library
  • sorangutan  

  1
  1
  Posts
  84
  Views

  

  Recently our friends at Figma announced their new Figma platform, and we’ve been really excited about its potential. We’ve immediately put the platform to use with Octicons, our SVG icon library.

  Distributing design assets effectively

  Previously, we checked our asset files into the GitHub repository. This workflow was restrictive and confusing for contributors who might want to iterate on or update an Octicon. We wanted anyone to be able to make contributions, but they needed all of these things to work in order to contribute.

  • Specific software: We were keeping icons in software specific to macOS. The contributor needed to have this software installed on the same version pinned in the Octicons repository.
  • Experience with Git: Contributors needed to clone the repository, edit the design asset, and check it back in. This requires knowledge of Git and how to get out of trouble when the binary file won’t merge properly.
  • Prior repository setup: After a contributor made changes they needed to run specialized scripts on their computer to output assets. This required all compatible versions of software installed properly on their laptop.

  Figma to the rescue

  To support your project’s contributors it’s important to make the contributing experience as frictionless as possible. Migrating our Octicons to Figma let us cut out painful steps in our previous workflow. Having their API available for automating the work has allowed contributors to contribute using powerful platform-agnostic design tools without any overly complex setup.

  Getting robots to do the work

  Robots are great for doing repeatable tasks, and handing that work off to automated systems frees us up to think about the big picture. We lean on continuous integration to build, export, and distribute the icons.

  Continuous integration (CI)

  On every pull request we use CI to export our icons from the file and distribute alpha versions of the libraries.

  

  Before and after images

  We also take advantage of Probot, an out-of-the-box robot that makes automating GitHub tasks easy.

  Probot has the ability to check our pull requests on Octicons and look for changes in the Figma source URL. When this occurs, our Probot app will query Figma’s platform and look for changes to any of the icons. When it finds those changes, it will comment on the pull request with before and after images. This makes the process easier for both contributors and maintainers.

  

  Excited about the future

  The API’s potential is the most exciting part, and we can’t wait to see how it improves our workflow. The Design Systems Team at GitHub are designers with an engineering focus. We want to keep our components in code, then make them available for our designers to prototype.

  The upcoming Figma write API will allow us to maintain our component library in code and export those as Figma components. Using a team library we can publish updates and make them available to the GitHub Product Design Team to use in their design mockups and prototypes.

  
  
  https://blog.github.com/2018-04-12-driving-changes-from-designs/
• 

  Four years of the GitHub Security Bug Bounty
  • sorangutan  

  1
  1
  Posts
  70
  Views

  

  Last month GitHub celebrated the fourth year of our Security Bug Bounty program. As we’ve done in the past, we’re sharing some details and highlights from 2017 and looking ahead to where we see the program going in 2018.

  2017 in review

  Last year was our biggest year yet as our Bug Bounty program continued to grow in participation by researchers, program initiatives, and the rewards paid out.

  Diving straight into the numbers, we can review the details of this growth. In 2017, we reviewed and triaged a total of 840 submissions to the program. Of these submissions, we resolved and rewarded a total of 121 reports with an average payout of $1,376 (and swag!). Compared to our previous statistics for 2016, this was a significant increase from 48 out of 795 reports being resolved. In 2017, our rate of valid reports increased from 6% to almost 15%.

  Our total payouts also saw a significant increase from $95,300 in 2016 to $166,495 in 2017. We attribute this to the increased number of valid reports and in October we took time to re-evaluate our payout structure. Corresponding with HackerOne’s Hack the World competition, we doubled our payout amounts across the board, bringing our minimum and maximum payouts to $555 and $20,000, bringing our bug bounty in line with the industry’s top programs.

  2017 initiatives

  To accelerate our program’s growth in 2017, we launched a number of initiatives to help engage researchers. Among the changes to the program was the introduction of GitHub Enterprise to the scope of the Bug Bounty program, which allowed researchers to focus on areas of our applications that may not be exposed on GitHub.com or are specific to certain enterprise deployments. In the beginning of 2017, a number of reports impacting our enterprise authentication methods prompted us to not only focus on this internally, but also identify how we could engage researchers to focus on this functionality. To promote a more targeted review of these critical code paths we kicked off two new initiatives beyond our public Bug Bounty program.

  Researcher grants

  Providing researcher grants is something that has been on our radar since Google launched their Vulnerability Research Grants in 2015. The basic premise is that we pay a fixed amount to a researcher to dig into a specific feature or area of the application. In addition to the fixed payment for the grant, any vulnerabilities identified would also be paid out through the Bug Bounty program. During the beginning of the year, we identified a researcher with specialty in assessing troublesome enterprise authentication methods. We reached out and launched our first researcher grant. We couldn’t have been happier with the results. It provided a depth of expertise and review that was well worth the extra monetary incentive.

  Private bug bounty

  In March 2017 we launched GitHub for Business, bringing enterprise authentication to organizations on GitHub.com. We used this feature launch as an opportunity to roll out a new part of the Bug Bounty program: private bug bounties. Through a private program on HackerOne, we reached out to all researchers who had previously participated in our program and allowed them access to this functionality before its public launch. This added to our internal pre-ship security assessments with review by external researchers and helped us identify and remediate issues before general exposure. With the extra review, we were able to limit the impact of vulnerabilities in production while also providing fresh code and functionality for researchers to look into.

  Operational efficiency

  Internal improvements to the program have helped us more efficiently triage and remediate submissions from researchers. ChatOps and GitHub-based workflows are core to how we deal with incoming submissions. As soon as new ones arrive, we receive alerts in Slack using HackerOne’s Slack integration. From there, we can triage issues directly from chat, letting the team know which issues are critical and which can wait until later. At the end of our triage workflow, we use ChatOps to issue rewards through HackerOne, so we can close the loop and pay researchers as quickly as possible.

  To support these workflows, we’ve continued to build on our Ruby on Rails HackerOne API client and extensively use these and GitHub APIs in our internal processes.

  So far, these improvements have made us significantly more efficient. Our average response time in 2017 was 10 hours, valid issues were triaged to developers on average within two days, and bounties were rewarded on average in 17 days. Given the time and effort that researchers dedicate to participating in our program, we feel great about these improvements. And in 2018, we’ll continue to refine our process. We’re always looking for ways to make sure our researchers receive a prompt and satisfactory response to their submissions.

  What’s next?

  Also in 2018, we’re planning to expand the initiatives that proved so successful last year. We’ll be launching more private bounties and research grants to gain focus on specific features both before and after they publicly launch. Later in the year, we’ll announce additional promotions to continue to keep researchers interested and excited to participate.

  Given the program’s success, we’re also looking to see how we can expand its scope to help secure our production services and protect GitHub’s ecosystem. We’re excited for what’s next and look forward to triaging and fixing your submissions this year!

  
  
  https://blog.github.com/2018-03-14-Four-years-of-bug-bounty/